High definition surveillance image storage optimization and retention triggering apparatus

ABSTRACT

A high definition surveillance video storage optimization system executes purging directives according to policy and retention meta data. Segments of video are purged from storage when thresholds of retention metrics are applied and when policy permits. Retention metrics are determined from acoustic sound energy, radiation energy, access control events, and movement. Policies reflect calendars and elapsed time, percentage of available storage, statutory and judicial mandates, and utility of the stored segments. Extrinsic sensor data and operations are recorded into non-transitory retention meta data stores. Metrics are transformed into retention flags at either time of recordation or time of purgation. Security file storage includes metadata about the environment acquired by sensors include doors, audio noise, object identification and tracking and inputs which influence retention or purging. A circuit filters for motion and stores metadata per segment. Meta data enables content based purging of least consequential files.

CROSS-REFERENCES TO RELATED APPLICATIONS

This non-provisional application is a continuation in part applicationof Ser. No. 15/338,507 Filed: Oct. 31, 2016 incorporated in its entiretywhich is in turn a CIP of Ser. No. 15/338,327 filed 29 Oct. 2016Content-driven surveillance image storage optimization apparatus andmethod of operation which is incorporated by reference in its entiretyand claims the benefit of provisional application 61/949,526 and itspriority date Mar. 7, 2014.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

THE NAMES OF THE PARTIES TO A JOINT RESEARCH AGREEMENT

Not Applicable

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISK OR ASA TEXT FILE VIA THE OFFICE ELECTRONIC FILING SYSTEM (EFS-WEB)

Not Applicable

STATEMENT REGARDING PRIOR DISCLOSURES BY THE INVENTOR OR A JOINTINVENTOR

Not Applicable

BACKGROUND OF THE INVENTION Technical Field

As is known, digital cameras record, store, and transmit images inseveral industry standard formats, such as H.234 and JPEG. The field ofthe invention is in the field of image analysis, and storageoptimization. The invention improves the operation of video retentionequipment.

Conventional file archive storage systems conserve space by time basedpurging of files by date of recording or by removing the larger fileswhich have not been used in recent time. Thus it can be appreciated thatwhat is needed is an improved system to more analytically selectablydelete only by content or meta data after an adjustable time period ofdays the video that does not provide significant content to save storagespace. In some cases, inactivity at a time and one or more locations issignificant. Triggering retention on motion alone may be an error.

Security Cameras

As is known, High Definition (HD) cameras used for security surveillance(SECCAMs) can produce high resolution and low resolution digital filesacross a wide spectrum both visible and invisible to the human eye. Thefiles may be of static images or moving images. The cameras may autofocus and auto adjust for lighting. The cameras may store data or pushit to network attached servers. Many SECCAMs do not pan or zoom andalways present the same view. Some SECCAMs take a still image whentriggered by a sensor, a timer, or an event. As is known, conventionalSECCAMs have hardware circuits to produce H.264 compatible video forhigh definition files.

As is known H.264 encoders vary the bit rate to dedicate more bits tothe frames that need them. Inter coding uses motion vectors forblock-based inter prediction to exploit temporal statisticaldependencies between different pictures. Intra coding uses variousspatial prediction modes to exploit spatial statistical dependencies inthe source signal for a single picture. Motion vectors and intraprediction modes may be specified for a variety of block sizes in thepicture. A motion vector is defined as “motion vector: A two-dimensionalvector used for inter prediction that provides an offset from thecoordinates in the decoded picture to the coordinates in a referencepicture.”

As is known, H.264 pictures include I frames, B frames and P frames.Pixel blocks of the I frames are referenced by P frames along withMotion Vectors. Motion vectors are encoded at the macroblock level in Por B frames. Motion vectors are encoded differentially relative toprevious motion vectors encoded in the frame. Furthermore, the syntaxand scale of the motion vectors depends on information that is includedin the picture header and picture coding extension header. After amacroblock has been decoded, the motion vector decoder can be queried inorder to get the current value of any motion vector. This information isno longer available once the next macroblock is decoded because it isreplaced by whatever motion vector information is associated with thenext macroblock instead. To capture motion vectors, you need to collectthem from the motion vector decoder after each macroblock is decoded.

jPEG

As is known, the name “jPEG” stands for joint Photographic ExpertsGroup, the name of the committee that created the jPEG standard and alsoother still picture coding standards.

The jPEG standard specifies the codec (code decode transform), whichdefines how an image is compressed into a stream of bytes anddecompressed back into an image, but not the file format used to containthat stream.

jPEG uses a lossy form of compression based on the discrete cosinetransform (DCT). This mathematical operation converts each frame/fieldof the video source from the spatial (2D) domain into the frequencydomain (aka transform domain.) A perceptual model based loosely on thehuman psychovisual system discards high-frequency information, i.e.sharp transitions in intensity, and color hue. In the transform domain,the process of reducing information is called quantization. In simplerterms, quantization is a method for optimally reducing a large numberscale (with different occurrences of each number) into a smaller one,and the transform-domain is a convenient representation of the imagebecause the high-frequency coefficients, which contribute less to theover picture than other coefficients, are characteristicallysmall-values with high compressibility. The quantized coefficients arethen sequenced and losslessly packed into the output bitstream. Nearlyall software implementations of JPEG permit user control over thecompression-ratio (as well as other optional parameters), allowing theuser to trade off picture-quality for smaller file size.

JPEG File Interchange Format (JFIF) is a minimal file format whichenables PEG bitstreams to be exchanged between a wide variety ofplatforms and applications. This minimal format does not include any ofthe advanced features found in the TIFF JPEG specification or anyapplication specific file format. Nor should it, for the only purpose ofthis simplified format is to allow the exchange of JPEG compressedimages.

JPEG Codec Example

Although a JPEG file can be encoded in various ways, most commonly it isdone with JFIF encoding. The encoding process consists of severalsteps: 1. The representation of the colors in the image is convertedfrom RGB to Y′CBCR, consisting of one luma component (Y′), representingbrightness, and two chroma components, (CB and CR), representing color.This step is sometimes skipped. 2. The resolution of the chroma data isreduced, usually by a factor of 2. This reflects the fact that the eyeis less sensitive to fine color details than to fine brightness details.3. The image is split into blocks of 8.times.8 pixels, and for eachblock, each of the Y, CB, and CR data undergoes the Discrete CosineTransform (DCT. A DCT is similar to a Fourier transform in the sensethat it produces a kind of spatial frequency spectrum. 4. The amplitudesof the frequency components are quantized. Human vision is much moresensitive to small variations in color or brightness over large areasthan to the strength of high-frequency brightness variations. Therefore,the magnitudes of the high-frequency components are stored with a loweraccuracy than the low-frequency components. 5. The resulting data forall 8.times.8 blocks is further compressed with a lossless algorithm, avariant of Huffman encoding. 6. The decoding process reverses thesesteps, except the quantization because it is irreversible.

DCT

As is known, the discrete cosine transform (DCT) expresses a finitesequence of data points in terms of a sum of cosine functionsoscillating at different frequencies. DCTs are important to numerousapplications in science and engineering, from lossy compression of audio(e.g. MP3) and images (e.g. JPEG) (where small high-frequency componentscan be discarded), to spectral methods for the numerical solution ofpartial differential equations. The use of cosine rather than sinefunctions is critical in these applications: for compression, it turnsout that cosine functions are much more efficient (fewer functions areneeded to approximate a typical signal), whereas for differentialequations the cosines express a particular choice of boundaryconditions.

In particular, a DCT is a Fourier-related transform similar to thediscrete Fourier transform (DFT), but using only real numbers. DCTs areequivalent to DFTs of roughly twice the length, operating on real datawith even symmetry (since the Fourier transform of a real and evenfunction is real and even), where in some variants the input and/oroutput data are shifted by half a sample. There are eight standard DCTvariants, of which four are common.

The most common variant of discrete cosine transform is the type-II DCT,which is often called simply “the DCT”, its inverse, the type-Ill DCT,is correspondingly often called simply “the inverse DCT” or “the IDCT”.

The compression method is usually lossy, meaning that some originalimage information is lost and cannot be restored, possibly affectingimage quality.

Like any Fourier-related transform, discrete cosine transforms (DCTs)express a function or a signal in terms of a sum of sinusoids withdifferent frequencies and amplitudes. Like the discrete Fouriertransform (DFT), a DCT operates on a function at a finite number ofdiscrete data points. The obvious distinction between a DCT and a DFT isthat the former uses only cosine functions, while the latter uses bothcosines and sines (in the form of complex exponentials).

Huffman Encoding/Decoding

As is known, Huffman coding is an entropy encoding algorithm used forlossless data compression. The term refers to the use of avariable-length code table for encoding a source symbol (such as acharacter in a file) where the variable-length code table has beenderived in a particular way based on the estimated probability ofoccurrence for each possible value of the source symbol.

Huffman coding uses a specific method for choosing the representationfor each symbol, resulting in a prefix code that expresses the mostcommon source symbols using shorter strings of bits than are used forless common source symbols. Huffman was able to design the mostefficient compression method of this type: no other mapping ofindividual source symbols to unique strings of bits will produce asmaller average output size when the actual symbol frequencies agreewith those used to create the code.

For a set of symbols with a uniform probability distribution and anumber of members which is a power of two, Huffman coding is equivalentto simple binary block encoding, e.g., ASCII coding. Huffman coding issuch a widespread method for creating prefix codes that the term“Huffman code” is widely used as a synonym for “prefix code” even whensuch a code is not produced by Huffman's algorithm.

Security Cameras

As is known, cameras used for security surveillance (SECCAMs) canproduce high resolution and low resolution digital files. The files maybe of static images or moving images. The cameras may auto focus andauto adjust for lighting. The cameras may store data or push it tonetwork attached servers. Many SECCAMs do not pan or zoom and alwayspresent the same view. Some SECCAMs take a still image when triggered bya sensor, a timer, or an event. As is known, conventional SECCAMs havehardware circuits to produce H.264 compatible video for high definitionas well as JPEG compatible encoded files. As is known, both software andhardware products to generate a stream of JPEG images from H.264 videoare commercially offered.

One problem with conventional SECCAMs installed with onsite storage isthat the data may be lost or stolen in the same incident for which theimages would have been useful. SECCAMs may provide timestamps at thecamera to record events for later viewing. In many cases, there is noattendant to monitor SECCAMs in real time. Thus economical remotestorage and delayed analysis are advantageous if the useful images couldbe transported without congesting a low bandwidth network. A requirementis that allocated bandwidth schedules not be exceeded. Within thispatent application we refer to JPEG compatible image files originatingor derived from SECCAMs as “previews” or “frames”.

BRIEF DESCRIPTION OF DRAWINGS

To further clarify the above and other advantages and features of thepresent invention, a more particular description of the invention willbe rendered by reference to specific embodiments thereof which areillustrated in the appended drawings. It is appreciated that thesedrawings depict only typical embodiments of the invention and aretherefore not to be considered limiting of its scope. The invention willbe described and explained with additional specificity and detailthrough the use of the accompanying drawings in which:

FIG. 1 is a block diagram of a conventional processor used forperforming method steps in an apparatus;

FIGS. 2-4 are flowcharts of steps in a method performed by a processor;

FIGS. 5-7 are block diagrams of an apparatus in a system;

FIG. 8-9 are block diagrams of an optimized surveillance image storageapparatus;

FIG. 10-12 are flowcharts of a method for purging surveillance imagestorage;

FIG. 13 is a block diagram of an apparatus,

FIG. 14 is a method flow chart, and FIGS. 15-16 are block diagrams ofstructural components of apparatus embodiments of a retention apparatus.

SUMMARY OF THE INVENTION

A high definition (HD) surveillance video storage optimization systemexecutes purging directives according to policy and retention meta data.Segments of video are purged from storage when thresholds of retentionmetrics are applied and when policy permits.

Retention metrics are determined from acoustic sound energy, radiationenergy, access control events, and movement. Measures of acoustic soundenergy are extracted from a HD recording or from sensors installed atthe surveillance location. Loud voices, impact sounds, sirens, andalarms may trigger setting a retention flag.

Radiation energy may be visible or non-visible spectrum extracted from aHD recording or infrared security sensors. Heat and temperature sensorsmay provide retention flags or metrics. Chemical sensors increase ordecrease conductivity through chamber. Accelerometers measure shaking,vibration, and physical movement.

Policies reflect calendars and elapsed time, percentage of availablestorage, statutory and judicial mandates, and utility of the storedsegments. Extrinsic sensor data and operations are recorded intonon-transitory retention meta data stores.

Metrics derived from the high definition video stream itself are storedinto files and file headers for a future purging process. Movementvectors of H.264 blocks and low frequency coefficients of pixel blocksare compared with a threshold to determine availability for purging.

Metrics are transformed into retention flags at either time ofrecordation or time of purgation

A storage system can either do motion analysis at purge time or when thevideo is initially archived. It is better if the motion analysis is doneduring the initial recording because the video can be pre-broken intosegments that contain motion and other segments that do not containmotion prior to being saved to storage. This would make it easier todelete the portions that do not contain motion at a time when freestorage is needed. Other analytic information could be added to theinput to this purge selection mechanism. Door opening and closinginformation could be used to determine to keep video. Security filestorage may include metadata about the environment acquired by sensorsas well. Ambient audio noise can be used to keep video as well.

Object identification and tracking could be used to keep the video aswell. Any number of video or external inputs could be used to decide tokeep the video. A content-driven purging circuit removes image segmentsfrom a surveillance image store according to purging directives incombination with retention meta data. An image transformation circuitprovides retention meta data based on movement metrics. The circuitdecodes and differences low frequency coefficients of correspondingpixel blocks in two image frames. Additionally, extrinsic measurementsand rules are embedded in directives and a retention meta data store.

A lossy compression method optimizes bandwidth and storage for asecurity surveillance network of appliances and event capture terminals.The method is performed by a processor upon an image file (Jframe)provided by a JPEG compatible encoder. The processor is adapted bycomputer readable instructions stored in non-transitory media to performoperations on Jframes in both the time domain and in the frequencydomain simultaneously which cannot be done mentally or by hand. Becausethe objective is security surveillance, the viewpoint of an eventcapture terminal neither pans nor zooms.

A method transforms a sequential plurality of Jframes into a traincomposed of a key frame (Kframe) and one or more subsequent frames(subframes). Each block of pixels in a key frame has coefficients in aplurality of frequency bins. Each corresponding block of pixels in asubsequent frame may contain the difference in coefficients between itssource Jframe and its Kframe. One or more subsequent frames follow a keyframe and are compressed relative to their untransformed Jframe bycontaining delta coefficients in the frequency domain. The resulting setof delta coefficients are re-encoded using a Huffman encoder.

Decompression combines a subsequent frame with its key frame to providean image with graduated resolution/noise clutter. While viewing asubsequent frame on its own is possible with a JPEG compatible browseror display it would seem to be meaningless without its reference keyframe. Reversing the transformation requires recovering all thefrequency coefficients of the key frame and the subframe and adding eachdelta coefficient to its corresponding Kframe coefficient. But theinvention further optimizes compression by removing thermal and visualnoise and sacrificing resolution in some of the pixel blocks. The methodfurther includes how to select pixel blocks and how to remove noise orsacrifice resolution.

A camera records, encodes, stores, and forwards a plurality of imagefiles, each file compatible with JPEG standard encoders/decoders(codecs). In addition to motion video or high definition or 3D or analogsignals, conventional cameras default to always providing a series ofstill images in digital format encoded by the JPEG standard. These arefrequency domain coefficients for blocks of pixels further encoded by aHuffman codec. A local area network such as Ethernet or Wi-Fi couplesthe camera with an appliance of the present disclosure.

Periodically or upon conditions in the content or external events, keyframes are selected from among the plurality of image files. Theapparatus may trigger on external events (e.g. sensors), image contents(e.g. motion, facial recognition), or internal processes such asperiodicity (e.g. timer, counter), or data stored (e.g. buffer capacity)to select a Jframe to serve as a key frame. The characteristics of onekey frame may influence the selection of a next key frame. The size of akey frame may influence the target size of its subsequent frames and theconfiguration of a low pass filter.

Meta data is stored separately or within the file headers of key framesand subsequent frames to enable decompression of a single subsequentframe. Meta data includes but is not limited to identification of keyframes and their locations (relative or absolute), the key frame eachsubsequent frame references, the size of files and the target size ortarget compression ratio, time and location, camera identity, and thelow pass filter configuration.

The transformation operates on coefficients of frequency bins. Oneaspect of the invention is to mask high frequency bins to achieve targetsize or compression ratios. Another aspect of the invention is to maskhigh frequency bins to remove thermal noise, lighting figments, andbackground vibration. Another aspect of the invention is to selectivelyimprove resolution of selected pixel blocks for pertinent securitysurveillance objectives.

A first frame in a stream of JPEG image frames is designated a first keyframe and stored. Each frame subsequent to the first key frame istransformed into a delta frame containing the differences from the firstkey frame. A second frame is designated a second key frame when themagnitudes of the differences cause the effective compression of themost recent transformed subsequent frame to fall below a threshold.

Data compression is obtained by only storing into non-transitory mediaand transmitting over a network the differences between a non-baselineframe and its baseline predecessor. Taking advantage of the nature ofsecurity cameras, there is high potential for successful compression.Many images taken are the same as the previous image. Even if an imagehas something new, most of the image is background or foreground, whichhas not changed. Security cameras are generally fixed and do not pan,zoom, or change in attitude. Lighting due to time of day will change andpersons or vehicles will enter, leave, or transit from one edge toanother.

A new baseline is selected below a target. Typically, the first frameafter a key frame has the least change and provides the best possiblecompression. The size of the difference files will range between thesize of the first frame and that of the key frame. A parameter is set todetermine a target minimum compression, which when exceeded causes a newkey frame to be selected.

By applying Huffman decoding, we recover coefficients for a DCTtransform of the image. These correspond to amplitudes for eachfrequency bin in a matrix of pixels. The amplitudes of a key frame arestored and the amplitudes of each subsequent image are subtracted fromthe corresponding coefficient. A small difference indicates little or nochange.

New key frames are selected either from a timer or counter or when thefollowing compression steps do not provide substantial savings. Thefirst compression is to compare a frame to its key frame and determineonly the differences. If the differences indicate little change, eventhe resolution can be reduced by zeroing high frequency data for furthercompression. Even significant differences in high frequency data may bezeroed if analysis of low frequency or DC data indicates the source isstatic noise or vibration. Advantageously, an image may be a mix ofhigher and lower resolution blocks if analysis of low frequency or DCdata indicates that a substantial change is occurring in that part ofthe image.

DETAILED DISCLOSURE OF EMBODIMENTS OF THE INVENTION

All of the following transformations and logic operations are performedby an electronic circuit. As is known, image file data may be read andstored to non-transitory computer readable media and operated on by aprocessor controlled by instructions in a store.

A lossy compression method optimizes bandwidth and storage for asecurity surveillance network of appliances and event capture terminals.Taking advantage of the nature of security cameras, there is highpotential for successful compression. Many images taken are the same asthe previous image. Even if an image has some new content, most of theimage is background or foreground, which has not changed. Securitycameras are generally fixed and do not pan, zoom, or change in attitude.Lighting due to time of day will change gradually and persons orvehicles will enter, progress from one edge to another, and exit thefield of view.

An event capture terminal (such as a camera) records, encodes, stores,and forwards a plurality of image files, each file compatible withstandard encoders/decoders. High Definition (HD) cameras can alsoproduce JPEG. H.264 video streams may be converted to a series of JPEGframes.

In one method a plurality of image frames are transformed into a trainmade of a key frame and one or more subsequent frames. By applyingHuffman decoding, we recover coefficients for a DCT transform of theimage. These correspond to amplitudes for each frequency bin in a matrixof pixels. The amplitudes of a key frame are stored and the amplitudesof each subsequent image are subtracted from the correspondingcoefficient of a block in its key frame.

Periodically or upon conditions in the content or external events, keyframes are selected from among the plurality of image files. New keyframes may be selected on a number of triggers. One could be an externalsensor, another from a timer or counter or file content such as when thefollowing compression steps do not provide substantial savings. Thefirst compression is to compare a frame to its key frame and utilize thedifferences in frequency coefficients. A further compression isavailable when the differences indicate little change, by zeroing highfrequency data. Further masking of significant differences in highfrequency data applies when analysis of low frequency or DC dataindicates the source is static noise or vibration. Advantageously forbandwidth and storage conservation, a resulting transformed image may bea mix of higher and lower resolution blocks if analysis of low frequencyor DC data indicates that a substantial change is localized to a portionof the full image.

The low pass filter is selectively applied to each pixel block within asubsequent frame. If there is substantial amplitude in DC or lowfrequency bins, it suggests movement into or out of the field of view.If there is no significant difference in the DC or low frequency bins,the coefficients in the high frequencies can be discarded without lossof information because they are likely to show only static, or noise, orvibration. Pixel blocks with DC and low frequency activity causeretention of the high frequency coefficients to provide a higherresolution image. An unchanging background can be lower resolutionwithout affecting the security value. So a transformed frame mayusefully have a mix of higher resolution pixel blocks and lowerresolution pixel blocks. As is known, DC originally refers to directcurrent and distinguishes a signal value from AC or alternating current.By extension, in the frequency domain it refers to the lowest range offrequencies in a spectrum.

One detailed aspect of the invention is a method for compression of aplurality of JPEG codec compatible files (Jframe) in time, spatial, andfrequency domains which has the steps: reversing Huffman encoding of afirst Jframe and storing every frequency coefficient of every block ofpixels in to a computer-readable store as a first key frame (Kframe);reversing Huffman encoding of an other Jframe to obtain every frequencycoefficient of every block of pixels; transforming the other Jframe intoa subsequent frame (subframe) by determining delta frequencycoefficients for each block of pixels in the subframe; storing all ofthe delta frequency coefficients of each block, and Huffman encoding thestored delta frequency coefficients of each block; when the resultingfile size meets the target compression goal, storing the file as asubsequent file and storing meta data relating the subsequent file toits related key file, and when the resulting file size exceeds thetarget compression goal, discarding the delta coefficients and storingthe original frequency coefficients of all the blocks as a new key file.

A delta coefficient is the difference between a frequency coefficient ofa block of pixels of a Jframe and the corresponding frequencycoefficient in its preceding Kframe.

This method performs compression in time by comparing a frame taken atone point in time with a reference frame from an earlier point in time.It also compresses spatially by operating on blocks of pixel in oneframe with the corresponding blocks of pixels in the reference frame.And the compression also operates in the frequency domain by subtractingeach DCT frequency coefficient from its equivalent.

Another process embodiment is a method for compression of a plurality ofJPEG codec compatible files (Jframe) in time, spatial, and frequencydomains which includes additional steps: reversing Huffman encoding of afirst Jframe and storing every frequency coefficient of every block ofpixels in to a computer-readable store as a first key frame (Kframe);reversing Huffman encoding of an other Jframe to obtain every frequencycoefficient of every block of pixels; transforming the other Jframe intoa subsequent frame (subframe) by determining delta frequencycoefficients for each block of pixels in the subframe; applying a motiontrigger rule to the delta frequency coefficients of each block of pixelsseparately; on determining a condition that the motion trigger rulepasses for each block of pixels, storing all of the delta frequencycoefficients of each block, and Huffman encoding the stored deltafrequency coefficients of each block; on determining a condition thatthe motion trigger rule fails for each block of pixels, storing zerosfor all the delta frequency coefficients of each block; when theresulting file size meets the target compression goal, storing the fileas a subsequent file and storing meta data relating the subsequent fileto its related key file, and when the resulting file size exceeds thetarget compression goal, discarding the delta coefficients and storingthe original frequency coefficients of all the blocks as a new keyframe.

A motion trigger rule may involve the DC or the lowest frequency rangeof bins for a block of pixels. Above a threshold, a change from thereference coefficients of the key frame may activate a motion triggerrule.

In an embodiment, the method performed by a processor coupled tonon-transitory computer readable storage further includes: applying anoise trigger rule to the delta frequency coefficients of each block ofpixel separately; on determining a condition that the noise trigger rulefails for each block of pixels, storing all of the delta frequencycoefficients of each block, and Huffman encoding the stored deltafrequency coefficients of each block; on determining a condition thatthe noise trigger rule passes for each block of pixels, applying a HighFrequency Coefficient (HFC) mask which zeros the H highest frequencydelta coefficients of each block, and Huffman encoding the HFC maskeddelta frequency coefficients; and, determining a file size of a filewhich contains all the Huffman encoded delta frequency coefficientswhich have been HFC masked or not masked.

A noise trigger rule may compare high frequency components to lowfrequency components with a block of pixels. When delta low frequencycomponents are substantially the same or below a threshold, while deltahigh frequency components are above a threshold, it may activate a noisetrigger rule.

In another embodiment, the method further includes: applying a leastsignificant bit (LSB) trigger rule to the delta frequency coefficientsof each block of pixel separately; on determining a condition that theLSB trigger rule fails for each block of pixels, storing all of thedelta frequency coefficients of each block, and Huffman encoding thestored delta frequency coefficients of each block; on determining acondition that the LSB trigger rule passes for each block of pixels,applying an LSB mask which zeros the L least significant bits of thedelta coefficients of each block, and Huffman encoding the LSB maskeddelta frequency coefficients; and, determining a file size of a filewhich contains all the Huffman encoded delta frequency coefficientswhich have been LSB masked or not masked.

Within each frequency bin the least significant bits may not beperceptible to a human observer. A LSB trigger rule may be tailored foran application or a venue or time of day or the type of display oranalysis. A facial recognition backend may require more bits than ahuman operator can use. The rule may set L the number of bits maskedoff.

In an embodiment, the method also includes: configuring a low passfilter; applying a configured low pass filter trigger (CLPF) rule to thedelta frequency coefficients of each block of pixels separately; ondetermining a condition that the CLPF trigger rule fails for each blockof pixels, storing all of the delta frequency coefficients of eachblock, and Huffman encoding the stored delta frequency coefficients ofeach block; on determining a condition that the CLPF trigger rule passesfor each block of pixels, applying the configured low pass filter to thedelta frequency coefficients of each block, and Huffman encoding thefiltered delta frequency coefficients; and, determining a file size of afile which contains all the Huffman encoded delta frequency coefficientswhich have been filtered or not filtered.

A configured low pass filter rule may be triggered by motion, or noise,or contents of the file, or by external events such as sensors, e.g.sound, vibration, temperature. Once configured, the CLPF may be appliedon a block by block basis according to its trigger rule.

In an embodiment, configuring a low pass filter includes the steps:reading a target compression ratio from a non-transitory computerreadable store; determining a target compressed file size by applyingthe target compression ratio to a first key frame; determining a numberP of highest frequency bins of delta frequency coefficients which whenzeroed, cause a first subsequent frame to meet the target compressedfile size; and storing P as the configuration of a low pass filter untila new key frame is selected.

In certain circumstances more than one trigger rule may be activated andmore than one compression method may be applied to a frame or even to ablock of pixels within a frame.

Another aspect of the invention is in the embodiment of circuits of afirst appliance locally attached to a network of event capture terminalstransforms and stores a plurality of image frames into a key frame andat least one subsequent frame. Meta data is stored separately or withinthe file headers of key frames and subsequent frames to enabledecompression of a single subsequent frame.

Referring now to the figures, a method embodiment is illustrated in FIG.2 to assist in apprehension of the subject matter. A method embodimentwhich is performed by a processor controlled by computer executableinstructions stored in non-transitory media includes: Receiving a KeyFrame (Kframe), storing size, determining target size for subsequentframes, and recovering baseline frequency bin coefficients for everypixel block by reversing Huffman encoding of a JPEG codec compatibleimage file (Jframe); 201 Several methods are disclosed infra to test aJframe for selection as a Key Frame. Once selected, its frequencycoefficients are used to determine delta coefficients for subsequentframes. Receiving 1st Jframe following Kframe 310; while unlikely due tothe nature of security surveillance systems, the method includesdetermining not another Kframe 312; Determining delta coefficients forevery frequency bin of every block of Jframe by reversing Huffmanencoding and subtracting each frequency bin coefficient from its Kframecorrespondent; 314 Configuring Low Pass Filter Mask to achieve targetsize for 1st subsequent frame 315 by setting an integer to denote thenumber of high frequency bins that are masked out up to a fixed minimumLow Pass Filter bandwidth (minLPF limit); Huffman encoding deltacoefficients to generate 1st Subsequent Frame (subFrame); 319

For each additional Jframe after the 1st subFrame until a next Kframe isselected 400: Reversing Huffman encoding to recover frequency bincoefficients for every block; 420 Determining delta coefficients forevery frequency bin of every pixel block of frame 430 by reversingHuffman encoding and subtracting each frequency bin coefficient from itscorrespondent in the Kframe. For each pixel block of the Jframe 500, oneof the following steps: Case I: When delta DC coefficient is greater orequal to a DC threshold, enabling all delta coefficients in the block;540 Case II: When delta DC coefficient is less than the DC threshold,and a sum of a group of high AC delta coefficients exceeds an ACthreshold, enabling only LPF unmasked delta coefficients in the block;550 Case III: When delta DC coefficient is less than the DC threshold,and a sum of a group of high AC delta coefficients is less than or equalto the AC threshold, forcing all delta coefficients in the block tozero; 560 When all pixel blocks of the Jframe have been transformed,Huffman encoding all the blocks to generate a next Subsequent Frame; 670which returns the process to step 400; Applying qualification test forselection as a Key Frame 780 which would return the process to step 201;and [0084] Storing meta data relating each Subsequent Frame (SubFrames)to a Key Frame in the file headers or in a separate table 890.

It can be appreciated that the sequential exposition of the method stepsis presented for clarity and not as a limitation precluding embodimentprocesses that may be efficiently performed in parallel or differentorder by circuits or processor threads.

Various methods can be employed to select a Key Frame from among aplurality of JPEG codec compatible image files. In an embodiment, amaximum period in time or in number of frames can be set as a parameter.When that maximum is met, an image is selected as a new key frame. Allthe frames following will be compared to the key frame until a new keyframe is selected.

One aspect of the invention is an exemplary apparatus having a circuitto receive JPEG codec compatible files and reverse Huffman encoding toobtain frequency coefficients for each pixel block; a circuit todistinguish key frames from subsequent frames; a circuit to set aconfigurable low pass filter for each train of a key frame and itssubsequent frame or frames; a circuit to determine delta coefficientsfor each frequency bin of each pixel block; a circuit to selectivelyzero insignificant delta coefficients; and a circuit to apply theconfigurable low pass filter to selected pixel blocks.

In an embodiment, the apparatus also has a circuit to Huffman encodedelta coefficients into a JPEG codec compatible file and store meta datato relate a subsequent frame to its key frame. In an embodiment, thecircuit to apply the configurable low pass filter is controlled by themagnitude of the delta coefficients in certain low frequency bins. In anembodiment, the circuit to set a configurable low pass filter determinesthe minimum number of high frequency coefficients to mask out to achievea target compression size.

Another aspect of the invention is an exemplary method for transformingJPEG codec compatible files by a processor communicatively coupled to atleast one event capture terminal and coupled to a non-transitorycomputer-executable instruction store, the method comprising: reading afirst target compression ratio store; receiving a first JPEG codeccompatible image file; determining a post-compression target file size;receiving a second JPEG codec compatible image file (Jframe); decodingfrequency bin coefficients for each pixel block of the first Jframe andof the second Jframe by reverse Huffman encoding; determining an integervalue to configure a low pass filter mask; determining a differencebetween each pair of respective coefficients of a pixel block in thefirst Jframe and its corresponding pixel block in the second Jframe;determining when the difference between each pair of respectivecoefficients of the low pass filter masked frequency bins is less than athreshold; and on the condition that the differences for all masked bitsare below the threshold; forcing the difference values of maskedfrequency bins to zero.

In an embodiment the process of determining an integer value toconfigure a low pass filter mask comprises: reading an integer value fora fixed minimum Low Pass Filter (minLPF) mask representing a limit to anumber of high frequency bins which may be set to zero, reading thepost-compression target file size; determining the smallest integervalue integer which successfully achieves the post-compression targetfile size; and storing the lesser of minLPF or the smallest integervalue which successfully achieves the post-compression target file sizeas the integer value to configure a low pass filter mask.

In an embodiment the method also includes determining that a 1st JPEGcodec compatible image file (Jframe) is a key frame by one ofperiodically or upon conditions in the content or external events,selecting a Jframe from among the plurality of image files by anexternal event, image content, periodicity in time or count, or quantityof data stored.

In an embodiment the method also includes storing metadata to identifythe relationship and location of key frames and subsequent frames.

In an embodiment the method also includes writing meta data into fileheaders for Kframes and subframes to identify their relationship.

In an embodiment the method also includes Huffman encoding thedifference values of all pixel blocks of the second Jframe to provide aJPEG compatible subsequent frame file.

Another aspect of the invention is an exemplary method for transformingJPEG codec compatible files by a processor communicatively coupled to atleast one event capture terminal and coupled to a non-transitorycomputer-executable instruction store, the method includes the steps:reading a first target compression ratio store; receiving a first JPEGcodec compatible image file; determining a post-compression target filesize; receiving a second JPEG codec compatible image file (Jframe);decoding frequency bin coefficients for each pixel block of the firstJframe and of the second Jframe by reverse Huffman encoding; selectingthe first Jframe as a key frame (Kframe) and the second Jframe as asubsequent frame (subframe); differencing coefficients of a key frameand a subsequent frame; re-encoding the difference in coefficients as aJPEG compatible subsequent frame; determining when the resultantsubsequent frame meets the required target size; and on the conditionthat the resultant subsequent frame meets the required target size,storing the file and annotating the meta data with the relationship ofthe subsequent file to the key file; or on the condition that theresultant subsequent frame fails to meet the required target size,discarding it and designating the second Jframe as a second key frame.

In an embodiment, the method also includes determining the absolutemagnitude of the difference between the coefficients of certain lowfrequency bins of the key frame and the subsequent frame; summing theabsolute magnitude of the differences between the coefficients ofcertain low frequency bins; on the condition when the sum of theabsolute magnitude of the coefficients of certain low frequency binsexceeds a threshold, enabling all of the differences betweencoefficients in all of the frequency bins to be available forre-encoding into a JPEG compatible file.

In an embodiment the method also includes determining the absolutemagnitude of the difference between the coefficients of certain lowfrequency bins of the key frame and the subsequent frame; summing theabsolute magnitude of the differences between the coefficients ofcertain low frequency bins; on the condition when the sum of theabsolute magnitude of the coefficients of certain low frequency bins isbelow a threshold, enabling only the coefficients of the low pass filtermask frequency bins for re-encoding into a JPEG compatible file.

In an embodiment the method also includes reading an integer value for afixed minimum Low Pass Filter (minLPF) mask representing a limit to anumber of high frequency bins which may be set to zero; reading thepost-compression target file size; determining the smallest integervalue integer which successfully achieves the post-compression targetfile size; storing the lesser of minLPF or the smallest integer valuewhich successfully achieves the post-compression target file size as theinteger value to configure a low pass filter mask; and designating thenon-masked bits of the low pass filter as indicators of certain lowfrequency bins.

In a useful application which illustrates the benefits and theprinciples of the subject of the application a local area networkcouples security cameras to a compression apparatus. It is desirablethat the long-term image storage be offsite to protect against theft ordestruction. Lower resolution image files in JPEG compatible fileformats are pushed across a public wide area network to a cloud storageservice. To avoid congestion, a maximum bit rate is allocated to thesecurity service at all times. The apparatus adaptively applies a set ofcompression rules and methods to meet the allocated budget for imagetransmission.

All the difference transformations are done in frequency space, whichmeans it only requires Huffman encoding/decoding of JPEG files which isvery fast. In addition, when motion detection is performed in DCT space,decoding all previews to this level can be done only once.

In an embodiment, the method further comprises when the differenceresults are high relative to a threshold; determining the image file isa new key file; storing the coefficients; and transmitting the imagefile to remote storage services as a new key file.

In an embodiment, the method further comprises when the differenceresults for low frequency coefficients is small and the differenceresults for high frequency coefficients are high, determining thepattern corresponds to static noise or vibration and setting the highfrequency difference results to zero; when the difference results forlow frequency coefficients is large and the difference for highfrequency coefficients are high, determining the pattern corresponds tomovement and storing the high frequency differences as computed.

In an embodiment, the method further comprises reading fromnon-transitory media a minimum threshold of compression parameter;determining a ratio of compression for each Huffman encoded differencefile to its key file; and on the condition that the ratio passes theminimum threshold, transmitting the difference file to remote storage,or on the condition that the ratio fails the minimum threshold,transmitting the image file to remote storage and storing itscoefficients as a new key file.

In an embodiment, the method further comprises when the differenceresults are high relative to a threshold; determining the image file isa new key file; storing the coefficients; and transmitting the imagefile to remote storage services as a new key file.

In an embodiment, the method further comprises when the differenceresults for low frequency coefficients is small and the differenceresults for high frequency coefficients are high, determining thepattern corresponds to static noise or vibration and setting the highfrequency difference results to zero; when the difference results forlow frequency coefficients is large and the difference for highfrequency coefficients are high, determining the pattern corresponds tomovement and storing the high frequency differences as computed.

In an embodiment, the method further comprises reading fromnon-transitory media a minimum threshold of compression parameter;determining a ratio of compression for each Huffman encoded differencefile to its key file; and on the condition that the ratio passes theminimum threshold, transmitting the difference file to remote storage,or on the condition that the ratio fails the minimum threshold,transmitting the image file to remote storage and storing itscoefficients as a new key file.

Reference will now be made to the drawings to describe various aspectsof exemplary embodiments of the invention. It should be understood thatthe drawings are diagrammatic and schematic representations of suchexemplary embodiments and, accordingly, are not limiting of the scope ofthe present invention, nor are the drawings necessarily drawn to scale.

Referring now to FIG. 3, another embodiment of the present invention isa method performed by a computer processor when executing instructionsstored in non-transitory computer readable media, the method fordetermining a compressed Jframe 200. By Jframe we mean an image filerecorded by a security surveillance camera and stored as a JPEG codeccompatible file. The method includes reading a stored minimumcompression threshold parameter 301; receiving an image frame from asecurity camera 310; Huffman decoding each frequency coefficient 330 foreach block of pixels in the image; determining a difference from a keyframe for each frequency coefficient 340 for every block; determining asubframe by Huffman encoding the difference in frequency coefficientsand computing total size 350; when a frame counter identifies the Kthsubframe following the previous Key frame, determining compression 360;when the Kth subframe meets the minimum compression threshold, storingthe size as a target 370; and when the subframe exceeds the minimumcompression threshold, select a new key frame 390.

Referring now to FIG. 4, an aspect of the invention is a method by aprocessor applying a test for selection as Key Frame 780. The methodincludes decrementing a Jframe counter or timer from the last key framewhich counts down to zero 410; receiving an image Jframe from a securitycamera 420; Huffman decoding each frequency coefficient 313 of a blockin the image Jframe; determining a difference from previous key framefor each frequency coefficient 314; Huffman encoding the differences infrequency coefficients 319; determining a storage size for thecompressed Jframe 460; when the compressed storage size exceeds a targetsize, storing the received image Jframe as a new key frame 470; storingall the frequency coefficients of the new key frame and the size of thenew key frame 480; and resetting Jframe counters to K+1 and timers toT+1 490.

Referring now to FIG. 5, an apparatus embodiment 500 of the presentinvention includes an image frame receiver 701 which receives JPEGcompatible files from a plurality of network attached securitysurveillance cameras (not shown). The image frame receiver is coupled toa subsequent frame store 710. The subsequent frame store is coupled to aHuffman decoder 720 which reverses part of the JPEG encoding process toyield coefficients for each block of pixels. The coefficients whichcorrespond to amplitudes of a DCT transformation are stored into theattached subsequent coefficient store 730. A key frame has previouslybeen selected, decoded, and stored. Its respective coefficients are readfrom a key coefficient store 740. Both stores are coupled to acoefficient differencer circuit 750 which subtracts the coefficients ofthe subsequent frame from the coefficients of the key frame. Thecoefficient differencer 750 is coupled to a low pass filter circuit 760which zeros the amplitudes of selected high frequency terms. That is,the high frequency components of a subsequent frame are discarded. Theremaining coefficients are Huffman encoded to create a JPEG compatiblefile but which only has the differences of certain coefficients betweenthe key frame and the subsequent frame. A key switch trigger 975 maydetermine that the degree of compression is satisfactory. The Huffmanencoded difference file is stored into a transmit buffer 779 foreventual uploading through a wide area network to a storage service.

Referring now to FIG. 6, an apparatus embodiment 600 of the presentinvention also includes a DC level trigger circuit 855 which is coupledto the output of the coefficient differencer 750, and to the input ofthe low pass filter 760. In an embodiment the DC level trigger is alsocoupled to the Huffman encoder 770 (not shown). Depending on variousstored parameters, also not shown, the DC level trigger enables ordisables the low pass filter 760 to zero selected high frequencycomponents. That is, the DC level trigger enables noise filtering on thecondition that the DC or low frequency components of the image are notsubstantially different from the key frame. When they are substantiallydifferent, a higher resolution block of pixels is passed to the Huffmanencoder 770. Beneficially, those blocks of the image which aresubstantially unchanging will have noise reduction and those blocks withsignificant change will have higher quality of resolution. Illustratedin FIG. 6 is the scenario in which the increased data of some blocks hasnot grown to a level which causes the key switch trigger 975 to activateso the resulting mixed resolution image is stored into the transmitbuffer 779.

Referring now to FIG. 7, an apparatus embodiment 700 of the presentinvention also includes a key image switch circuit 776 which is coupledto the subsequent frame store 710 and a key frame store 916, and iscoupled to the subsequent coefficient store 730 and the key coefficientstore 740. The key image switch 776 is controlled by a key switchtrigger circuit 975 which is coupled to the Huffman encoder 770. The keyframe store 916 is further coupled to a transmit buffer 779. A new keyframe may be selected for a number of reasons: there may be a counter ortimer to establish a maximum period for transmitting a key frame fromwhich all subsequent frames are differenced. But a beneficial aspect ofthe present invention is to adaptively select key frames to performcompression on a series of JPEG compatible image frames. In the exampleshown, the DC level trigger 855 determines that one or more blocks ofthe image have substantially changed and disables the low pass filter760 from zeroing high frequency coefficients. After the encoding of thedifference frame in the Huffman encoder 770, the size of the result andthe degree of compression attained is compared with a target size by thekey switch trigger 975. When the key switch trigger determines that thecompression is no longer sufficiently advantageous (according to storedparameters not shown), the key image switch selects the currentsubsequent frame as a new key frame by transferring the subsequent framefrom the subsequent frame store 710 into the key frame store 916 andalso to the transmit buffer 779. The contents of the subsequentcoefficient store 730 are also transferred to the key coefficient store740. Embodiments of the circuits may also read out the subsequent framestore 710 directly to a transmit buffer 779 as the following subsequentframe is read in. Circuit functionalities herein disclosed may becombined into more complex circuits or parallel processor instructionswithout changing the claimed invention.

An apparatus receives image frames from security video cameras. If theimages are already encoded using a JPEG compatible codec, they aretransformed by Huffman decoding to produce a plurality of blocks havingamplitudes as frequency coefficients.

Compression is controlled by a set of parameters which includesselection of key frames.

A change exceeding a threshold in lower-frequency amplitudes, determinedby a set of parameters, controls noise filtering. Changes in higherfrequency coefficients are not encoded when the size of images exceeds atarget.

After high-frequency amplitudes have been clipped to remove noise fromselected blocks, the remaining frequency amplitudes of the subsequentframe is compared with a key frame and a delta frame is determined. Theamount of compression achieved by the delta is computed and tracked.Each delta is transformed by Huffman encoding and stored as a JPEGunless the amount of compression is decreasing. On that condition, a newkey frame is selected.

One aspect of the present invention performs, by a processor, deltacompression of jpeg preview images. A key frame is stored innon-transitory media, and a subsequent frame is encoded as deltas fromthat key frame. The amount of compression being achieved is tracked, andwhen it starts decreasing a new key frame is generated. The delta framesare also encoded as jpeg compatible files, so they also benefit from theJPEG codec's compression/encoding gains. Thus, if one displays a rawdelta frame, it appears as an all grey preview because each block offrequency coefficients are difference values between coefficients of akey frame and coefficients of a subsequent image frame. For staticscenes, the embodiment has achieved about 4.times. compression. All ofthis is “unwound” within the archiver, so has no impact on externalAPIs.

All the difference transformations are done in frequency space, whichadvantageously requires only Huffman encoding/decoding of JPEG fileswhich is very fast. In addition, when motion detection is performed inDCT space, decoding all previews to this level can be done only once.

JPEG files are encoded in a “spatial frequency” dimension. At night,cameras normally move to very high gain settings, which creates“speckle” noise—each pixel has small relatively random noise added toit. This creates a lot of high frequency noise and cause JPEG (and anyother video compression scheme) to produce much larger images. In thepresent invention, the method arbitrarily “clips” this data, notencoding changes in the higher frequencies when it determines the sizestarting to grow out of control.

While this clipping it normally invisible, where motion is actuallyoccurring the clipping can significantly degrade the utility of theresulting image. To avoid this, if a JPEG block has significant changein the lower frequency coefficients the clipping is prevented. Contraryto privacy considerations, it is the home intruder who is kept inhighest definition while the pixel blocks of static room occupants arecompressed into lower resolution.

A system provides a cloud based archival and presentation apparatuscommunicatively coupled by an unreliable low bandwidth wide area networkto a transmission apparatus locally networked to a plurality ofconventional security video capture devices. The transmission apparatusreceives a sequence of images, determines motion, removes noise, selectskey frames, compresses interstitial frames, and transmits compressedjpeg previews to the archival and presentation apparatus.

One aspect of the invention is a method for operation of a systemperformed at processors executing computer-instructions encoded innon-transitory media on data stored in electronic circuits by: at atransmission apparatus coupled to a plurality of security video capturedevices, receiving image files split into blocks of 8.times.8 pixels,transformed by a Discrete Cosine Transform (DCT) into frequency bins. ADCT is similar to a Fourier transform in the sense that it produces akind of spatial frequency spectrum; decompressing the data bytransforming the image file components by applying a variant of Huffmanencoding; reading the amplitudes of the frequency components whichinclude high-frequency components, mid-frequency components, andlow-frequency components; determining motion; reading stored parameterswhich control noise suppression; suppressing noise components whendetermining motion is not true; reading stored parameters which controlselection of key frames; selecting a first key frame; comparing aplurality of subsequent frames to the first key frame; determining adelta frame between each subsequent frame and a first key frame;determining an amount of compression which each delta frame has over thefirst key frame, while the amount of compression is not decreasing foreach subsequent frame, encoding and storing the delta frame with JPEGtransformation; while the amount of compression is decreasing,discarding the delta frame and selecting a second key frame; andtransmitting key frames and delta frames to an archival apparatus.

The techniques described herein can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations of them. The techniques can be implemented as a computerprogram product, i.e., a computer program tangibly embodied in aninformation carrier, e.g., in a machine-readable storage device or in apropagated signal, for execution by, or to control the operation of,data processing apparatus, e.g., a programmable processor, a computer,or multiple computers. A computer program can be written in any form ofprogramming language, including compiled or interpreted languages, andit can be deployed in any form, including as a stand-alone program or asa module, component, subroutine, or other unit suitable for use in acomputing environment. A computer program can be deployed to be executedon one computer or on multiple computers at one site or distributedacross multiple sites and interconnected by a communication network.

Method steps of the techniques described herein can be performed by oneor more programmable processors executing a computer program to performfunctions of the invention by operating on input data and generatingoutput. Method steps can also be performed by, and apparatus of theinvention can be implemented as, special purpose logic circuitry, e.g.,an FPGA (field programmable gate array) or an ASIC (application-specificintegrated circuit). Modules can refer to portions of the computerprogram and/or the processor/special circuitry that implements thatfunctionality.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for executing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. Information carrierssuitable for embodying computer program instructions and data includeall forms of non-volatile memory, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory can be supplemented by, or incorporated in special purposelogic circuitry.

FIG. 1 is a block diagram of an exemplary processor that may be used toperform one or more of the functions described herein. Referring to FIG.1, processor 100 may comprise an exemplary client or server process.Processor 100 comprises a communication mechanism or bus 111 forcommunicating information, and a processor core 112 coupled with bus 111for processing information. Processor core 112 comprises at least oneprocessor core, but is not limited to a processor core, such as forexample, ARM™, Pentium™, etc.

Processor 100 further comprises a random access memory (RAM), or otherdynamic storage device 104 (referred to as main memory) coupled to bus111 for storing information and instructions to be executed by processor112. Main memory 104 also may be used for storing temporary variables orother intermediate information during execution of instructions byprocessor core 112.

Processor 100 also comprises a read only memory (ROM) and/or otherstatic storage device 106 coupled to bus 111 for storing staticinformation and instructions for processor core 112, and anon-transitory data storage device 107, such as a magnetic storagedevice or flash memory and its associated control circuits. Data storagedevice 107 is coupled to bus 111 for storing information andinstructions.

Processor 100 may further be coupled to a display device 121 such a flatpanel display, coupled to bus 111 for displaying information to acomputer user. Voice recognition, optical sensor, motion sensor,microphone, keyboard, touch screen input, and pointing devices 123 maybe attached to bus 111 or a wireless network interface 125 forcommunicating selections and command and data input to processor core112.

Referring to FIGS. 8-12, one aspect of the content-driven storageoptimization invention is a content-driven surveillance storageoptimization apparatus 800 of FIG. 8 which includes: a purging circuit831; a surveillance image transformation circuit 861; a surveillanceimage store 852 communicatively coupled to the purging circuit and tothe surveillance image transformation circuit; a retention meta datastore 854 communicatively coupled to the purging circuit and to thesurveillance image transformation circuit; and a triggers, inhibitors,policies controller 811 communicatively coupled to the purging circuit,whereby transformed content of the surveillance images providesretention meta data to be used by the purging circuit when activated bythe controller to select images for deletion. In an embodiment, thesurveillance image store is directly coupled to at least onesurveillance image capture unit 871-879. In an embodiment, thesurveillance image transformation circuit is directly coupled to atleast one surveillance image capture unit 871-879. In an embodiment,transformation includes the step of producing a stream of JPEGcompatible images from an H.264 protocol of high definition video.

If there is substantial amplitude in DC or low frequency bins, itsuggests movement into or out of the field of view. As is known, DCoriginally refers to direct current and distinguishes a signal valuefrom AC or alternating current. By extension, in the frequency domain itrefers to the lowest range of frequencies in a spectrum.

A surveillance image transformation circuit includes an image framereceiver which receives JPEG compatible files from a plurality ofnetwork attached security surveillance cameras. The image frame receiveris coupled to a subsequent frame store. The subsequent frame store iscoupled to a Huffman decoder which reverses part of the JPEG encodingprocess to yield coefficients for each block of pixels. The coefficientswhich correspond to amplitudes of a DCT transformation are stored intothe attached subsequent coefficient store. A key frame has previouslybeen selected, decoded, and stored. Its respective coefficients are readfrom a key coefficient store. Both stores are coupled to a coefficientdifferencer circuit which subtracts the coefficients of the subsequentframe from the coefficients of the key frame.

The coefficient differencer is coupled to a DC level trigger whichdetermines that one or more blocks of the image have substantiallychanged and records a metric into retention meta data store.

A motion measure circuit involves the DC or the lowest frequency rangeof bins for a block of pixels. Above a threshold defined in a purgingdirective, a magnitude of change from the reference coefficients of thekey frame may set a retention flag. The purging directive may includeother triggers or policies to control retention. Thresholds may be setfor sensor energy, for age relative to recordation or usage, andcombinations of age and magnitude of movement. A store of policies andrules such as injunctions is coupled to a controller. The purgingcircuit includes a circuit to read retention meta data, a circuit toread purging directives, and a circuit to set a retention flag byapplying the directives to retention meta data. The purging circuitissues file operation commands according to the result of matchingretention meta data with thresholds, triggers, and policies of thepurging directives.

In an embodiment, the controller triggers on free space and least ormost recently used file I/O. In an embodiment, the controller triggerson calendars, policies, court orders, and statutory requirements. In anembodiment illustrated in FIG. 9, the apparatus 900 also includes aretention meta data combiner circuit 991, coupled to the retention metadata store 854, and coupled to the surveillance image transformationcircuit 861; whereby metrics received from extrinsic sensors 981-989 arecorrelated and combined into retention meta data and stored into theretention meta data store to influence the purging circuit. In variousembodiments, extrinsic sensor metrics are at least one of the group:sound energy, vibration energy, physical access control events, andthermal energy.

Other extrinsic sensors include chemistry, heart rate, and bio-metrics.

As illustrated in FIG. 10, the surveillance storage optimization method1000 includes: receiving one or more purging control directives 1050;retrieving a retention meta data array for each of a plurality of imagesegment identifiers 1060; masking each retrieved retention meta dataarray by each received purging control directive to set a retention flagfor each image segment identifier 1070; and purging from surveillanceimage store, the image segments corresponding to image segmentidentifiers not set for retention 1090. In an embodiment, a retentionmeta data array includes a metric of relative movement within the imagesegment. In an embodiment of the invention, the method further includestransformation step of image frames including: receiving surveillanceimage frames from capture devices 1010; storing captured surveillanceimage frames into a surveillance image store 1021; transforming storedcaptured surveillance image frames into image segments and retentionmeta data arrays 1023; and storing image segment identifiers andretention meta data arrays into a retention meta data store 1040.

In an other embodiment, illustrated in FIG. 11 the transformation stepsinclude receiving surveillance image frames from capture devices 1010;transforming captured surveillance image frames into image segments andretention meta data arrays 1125; storing transformed image segments intosurveillance image store 1127; receiving extrinsic sensor measurements1132; combining extrinsic sensor measurements into retention meta dataarrays 1134; and storing image segment identifiers and retention metadata arrays into a retention meta data store 1040.

The inventive image compression and optimization invention disclosedabove also contributes to retention meta data in the following way:

Another process embodiment is a method for compression of a plurality ofJPEG codec compatible files (Jframe) in time, spatial, and frequencydomains which includes additional steps: reversing Huffman encoding of afirst Jframe and storing every frequency coefficient of every block ofpixels in to a computer-readable store as a first key frame (Kframe);reversing Huffman encoding of an other Jframe to obtain every frequencycoefficient of every block of pixels; transforming the other Jframe intoa subsequent frame (subframe) by determining delta frequencycoefficients for each block of pixels in the subframe; applying a motiontrigger rule to the delta frequency coefficients of each block of pixelsseparately; on determining a condition that the motion trigger rulepasses for each block of pixels, storing all of the delta frequencycoefficients of each block, and Huffman encoding the stored deltafrequency coefficients of each block; on determining a condition thatthe motion trigger rule fails for each block of pixels, storing zerosfor all the delta frequency coefficients of each block; when theresulting file size meets the target compression goal, storing the fileas a subsequent file and storing meta data relating the subsequent fileto its related key file, and when the resulting file size exceeds thetarget compression goal, discarding the delta coefficients and storingthe original frequency coefficients of all the blocks as a new keyframe.

A motion trigger rule may involve the DC or the lowest frequency rangeof bins for a block of pixels. Above a threshold, a change from thereference coefficients of the key frame may be stored as a retentionmeta data and activate a motion trigger rule or deactivate a purgingdirective.

In an embodiment, a determination of a metric of relative movementbetween a first image frame and a second image from includes:determining the absolute magnitude of the difference between thecoefficients of certain low frequency bins of the key frame and thesubsequent frame; summing the absolute magnitude of the differencesbetween the coefficients of certain low frequency bins; and storing thesum into a retention meta data array as a metric of relative movement.

In an embodiment, a method for determining a metric of relative movementis illustrated in FIG. 12. The method 1200 includes the processes:receiving a key JPEG frame (Kframe) and a subsequent frame (Jframe)1210; reversing Huffman encoding of coefficients of pixel blocks 1230;differencing the low frequency coefficients between corresponding pixelblocks in Kframes and Jframes 1250; determining a metric of relativemovement by normalizing the difference of low frequency coefficients1270; and, storing into retention meta data store an image segmentidentifier and an associated metric of relative movement 1290.

In an embodiment, a retention meta data array includes an extrinsicsensor measurement correlated to the image segment. In an embodiment, aretention meta data array includes a timestamp of file creation andrecords of file system operations on the image segment.

In an embodiment, a retention meta data array includes location andownership of a capture device associated with an image segment.

In an embodiment, extrinsic sensor measurements include sound energy,vibration energy, thermal energy, and state of physical access controlportals.

In an embodiment, purging control directives include a threshold of ametric of relative movement within the image segment, a level ofextrinsic sensor measurement, and an inequality of at least one ofelapsed time since capture, most recent file access, and least recentfile access. In an embodiment, purging control directives includeownership, location, and time of capture as specified in judicialorders, legislative statutes, and administrative policies.

Referring to FIG. 13, another aspect of the invention is an apparatus1300 including: a purging module 1310; a network interface to extrinsicsensors 1320; an extrinsic retention meta data store 1330; a videostream transformation module 1340; and a network interface to aplurality of high definition video cameras 1350, said modules,interfaces, and interconnected by communication circuits and switches.In an embodiment, the apparatus also includes: a control panel to enterpurging directives 1360; and an interface to a non-transitory store ofvideo segment files 1370. In an embodiment, the apparatus also includes:a circuit to receive H.264 streams from high definition video cameras; acircuit to segment H.264 streams into files and embed retention metadata in each header of a file.

In an embodiment, the apparatus also includes: a circuit to determinethat an extrinsic sensor measurement or event is of substantialsignificance to a surveillance system; and, a circuit to set a retentionflag in an extrinsic retention meta data store.

In an embodiment, the apparatus also includes: a circuit to decodemovement vectors and low frequency coefficients in pixel blocks; and acircuit to set a movement flag into retention meta data of a videosegment file. In an embodiment, the apparatus also includes: a circuitto receive a purging directive; and a circuit to remove pointers tovideo file segment files without a retention flag set.

In an embodiment, said purging module 1310 includes: a retention policystore; a statutory and judicial requirement store; a metric thresholdstore; a calendar module; a camera and location ownership store; aninterface to disk-file management; and a circuit to score retention metadata. In an embodiment, said extrinsic retention meta data store 1330has, encoded in non-transitory media associated by time with videosegments, sensor measurements selected from the groups: portal transits;physical access control requests and actuations; occupancy of rooms,halls, and location space; vibration and movement from accelerometers;temperature and heat; wind speed and humidity; acoustical sound energy;electromagnetic signals from motion, smoke, fire, CO detectors; andperimeter or equipment intrusion alarms.

In an embodiment, said video transformation module 1340 comprises: acircuit to transform H.264 streams into video segments; a circuit toannotate video segments with metrics of acoustic energy transitions; acircuit to annotate video segments with metrics of visible andnon-visible spectrum energy transitions; a circuit to transform segmentsof H.264 video stream into metrics of movement vectors; a circuit totransform segments of H.264 video stream into JPEG pixel blocks; acircuit to transform frequency coefficients to movement metrics; and acircuit to store said metrics into file headers.

Referring to FIG. 14, another aspect of the invention is method ofoperation including: receiving a high definition video stream 1410;transforming the video stream into segments with retention meta dataheaders 1420; receiving extrinsic events and sensor measurements 1430;receiving retention policies and metric thresholds 1440; determiningretention metrics and flags 1450; receiving a purging directive 1460;evaluating calendar and policy constraints on purging 1470; maskingvideo file segments protected by retention meta data from purging 1480;and removing pointers to video file segments available to purging 1490.In an embodiment, receiving a high definition video stream 1410includes: connecting to a plurality of high definition cameras;downloading video streams in at least one of H.264 and JPEG coding; andreversing coding to obtain movement vectors or frequency bincoefficients. In an embodiment, transforming the video stream intosegments with retention meta data headers includes: selecting blocks oftime to segment a stream; converting a stream into video segment files;generating a plurality of low resolution preview frames; storinglocations of segment files and preview frames; determining motionmetrics from contents of segments and previews; and storing retentionmeta data into tables and file headers.

In an embodiment, receiving extrinsic events and sensor measurements1430 includes at least one of the group: receiving an indication ofportal transit; receiving an indication of area occupancy; receiving ameasurement of acoustic energy change; receiving a measurement ofvibration and movement; receiving a measurement of radiation energychange; receiving an access control request and actuation; receiving ameasurement of temperature or thermal change; receiving a building orequipment intrusion alarm; and receiving a chemical sensor measurement.

In an embodiment, receiving retention policies and metric thresholds1440 includes at least one of the group: receiving a minimum time fromrecordation to retain video; receiving a minimum time from most recentuse to retain video; receiving a minimum free space to trigger purging;receiving a minimum measure of movement to flag retention; receiving aperiodic time to trigger purging; receiving a statutory or judicialrequirement for retention; receiving a threshold of a sensor measurementfor retention; and receiving an extrinsic event to flag retention.

In an embodiment, determining retention metrics and flags 1450 includes:comparing low frequency coefficients with a threshold; comparing changesin heat, light, sound, and vibration with a threshold; comparingmagnitude of motion vectors with a threshold; comparing traffic andoccupancy with norms by calendar; and comparing extrinsic events withnorms by time and day.

In an embodiment, receiving a purging directive 1460 includes: receivinga command from a user interface; and receiving parameters on thresholds,dates, and locations.

In an embodiment, evaluating calendar and policy constraints on purging1470 includes: determining that time since recordation has passed aretention threshold; determining that time since last use has passed aretention threshold; determining that percentage of free space hasfallen below a threshold; determining that location of the cameras isnot within a retention zone; determining that ownership of the camerasis not by a retention client; and determining that there are no judicialor statutory constraints on purging video segments owned by the owner,during that time period, at the location of the camera.

In an embodiment, masking video file segments protected by retentionmeta data from purging 1480 includes: listing all video file segmentsolder than minimum retention from recordation; removing video filesegments recently used within a retention window; comparing retentionmeta data metrics with thresholds; setting retention flags on video filesegments with metrics of significant value; setting retention flags onvideo file segments which coincide with extrinsic events; extractingretention flags and metrics embedded in file headers; and, settingretention flags on file segments subject to legal requirements,statutes, warrants, court orders, and subpoenas.

In an embodiment, removing pointers to video file segments available topurging 1490 includes: determining when a file segment is protected frompurging by legal constraints; determining when a file segment isprotected from purging by policy; determining when a file segment isprotected from purging by elapsed time from recordation or use;determining when a file segment is protected from purging by anextrinsic retention meta data value or event mark; determining when afile segment is protected from purging by a measure of content above athreshold; and removing a pointer to unprotected video segments from thefile system whereby the non-transitory media is returned to free space.

Now referring to FIG. 15, another aspect of the invention is anapparatus for high definition surveillance image storage optimization1500 which includes:

a purging module 1510; anda network interface 1521 to extrinsic sensors; communicatively coupledto,an extrinsic retention meta data store 1530 coupled to the purgingmodule and further coupled to;a video stream transformation module (vtransformation module 1540); and,coupled to said vtransformation module,a camera network interface 1551 to a plurality of high definition videocameras, said modules, interfaces, and interconnected by communicationcircuits and switches.In an embodiment, the apparatus also includes:a control panel 1511 to enter purging directives; the control panelcommunicatively coupled to said purging module, and, communicativelycoupled to said vtransformation module,an interface 1561 to a non-transitory store of video segment files 1560.In an embodiment, the apparatus also includes:a circuit 1552 to receive H.264 streams from high definition videocameras;a circuit 1531 to segment H.264 streams into files and embed retentionmeta data in each header of a file.In an embodiment, the apparatus also includes:an apparatus for image storage optimization and retention includinga processor core 1522 configured to determine that an extrinsic sensormeasurement or event is of substantial significance to a surveillancesystem; and,a circuit 1532 to set a retention flag in an extrinsic retention metadata store.In an embodiment, the apparatus also includes:a circuit 1549 to decode movement vectors and low frequency coefficientsin pixel blocks; and a circuit to set a movement flag into retentionmeta data of a video segment file 1563.

Another aspect of the invention illustrated in FIG. 16, the apparatus1600 also includes: a circuit 1670 to receive a purging directive; and acircuit 1679 to remove pointers to video file segment files 1660 withouta retention flag set. In an embodiment, said purging module 1610includes: a retention policy store 1612; a statutory and judicialrequirement store 1613; a metric threshold store 1614; a camera andlocation ownership store 1615; an interface to disk-file percentage usemanagement 1616; and a circuit to score retention meta data 1617 and acalendar module 1680 comprising a calendar processor configured byinstructions to match video file segment time stamps with date-time ofpurging directives and when true, trigger purging of segmentsunprotected by retention meta data. In an embodiment, said extrinsicretention meta data store 1630 has, encoded in non-transitory mediaassociated by time with video segments, sensor measurements selectedfrom the groups: portal transits; physical access control requests andactuations; occupancy of rooms, halls, and location space; vibration andmovement from accelerometers; temperature and heat; wind speed andhumidity; acoustical sound energy; electromagnetic signals from motion,smoke, fire, CO detectors; and perimeter or equipment intrusion alarms.

In various embodiments, said video transformation module 1640 includes:at least one of a digital logic circuit 1642 to receive H.264 streamstransmitted from high definition video cameras coupled to said cameranetwork interface 1651; a first processor core 1643 configured byinstructions to transform a received H.264 stream into at least onevideo segment; communicatively coupled to, a second processor core 1644configured by instructions to annotate said video segment by insertionof at least one metric of acoustic energy transitions; communicativelycoupled to, a third processor core 1645 configured by instructions toannotate video segments by insertion of metrics of visible andnon-visible spectrum energy transitions; communicatively coupled to afourth processor core 1646 configured by instructions to transform videosegments of the received H.264 video stream into metrics of movementvectors of pixel blocks; communicatively coupled to, a fifth processorcore 1647 configured by instructions to transform video segments of thereceived H.264 video stream into JPEG pixel blocks; communicativelycoupled to a sixth processor core 1648 configured by instructions totransform frequency coefficients of said JPEG pixel blocks into movementmetrics; and, coupled to said processor cores 1643-48, a digital logiccircuit 1649 to transmit a file write operation to media controller tostore said metrics into file headers encoded into a non-transitory mediastorage device.

Another aspect of the invention is a method for surveillance imagestorage optimization having the processes: receiving one or more purgingcontrol directives; retrieving a retention meta data array for each of aplurality of image segment identifiers; masking each retrieved retentionmeta data array by each received purging control directive to set aretention flag for each image segment identifier; purging fromsurveillance image store, the image segments corresponding to imagesegment identifiers not set for retention; transforming high definitionvideo into a plurality of image frames; transforming stored image framesinto image segments and retention meta data arrays; storing imagesegment identifiers and retention meta data arrays into a retention metadata store; reversing Huffman encoding of coefficients of pixel blocks;differencing the low frequency coefficients between corresponding pixelblocks in Kframes and Jframes; determining a metric of relative movementby normalizing the difference of low frequency coefficients; and,storing into retention meta data store an image segment identifier andan associated metric of relative movement; wherein a retention meta dataarray includes an extrinsic sensor measurement correlated to the imagesegment; wherein a retention meta data array includes a timestamp offile creation and records of file system operations on the imagesegment; wherein purging control directives include a threshold of ametric of relative movement within the image segment, a level ofextrinsic sensor measurement, and an inequality of at least one ofelapsed time since capture, most recent file access, and least recentfile access; and wherein purging control directives include ownership,location, and time of capture as specified in judicial orders,legislative statutes, and administrative policies.

CONCLUSION

The present invention can be easily distinguished from time or sizebased conventional file management systems by its recording/storingsurveillance images while determining image movement meta dataparameters (e.g. low frequency coefficients of JPEG pixel blocks);combining extrinsic records into meta data with image/movement meta data(e.g. sound energy in video recording); setting purgingtriggers/inhibitors/policies; purging triggers/inhibitors circuit whichimplements: an administrative record retention policy, statutoryregulations, judicial orders; and is triggered by time since most recentuse, time since recordation/storage, % of available free space on media,and relative range of purging meta data parameters. An exemplarydirective could include: “Purge image segments below thresholds of metadata older than _(— — — — — —) upon _(— — — — — —) ”

Content and extrinsic meta data parameters include: movement range, lowfrequency bins of pixel blocks, rate of change in low frequencycoefficients, and out of norm for time of day/day of week. Retentioncontrol by access control events include door opening, window opening,and traffic/volume measurements. Retention control by environmentalsensors include sound energy at microphones, thermal energy at sensors,vibration energy at accelerometers, heart rate/blood pressure/chemistry,and heating/ac/light switches.

A number of embodiments of the invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention. Forexample, other network topologies may be used. Accordingly, otherembodiments are within the scope of the following claims.

1-3. (canceled)
 4. An apparatus for image storage optimization andretention comprising: a purging module; communicatively coupled to, anetwork interface to extrinsic sensors; communicatively coupled to, anextrinsic retention meta-data store; coupled to the purging module andfurther coupled to a camera network interface to a plurality of highdefinition video cameras; said modules interfaces and storeinterconnected by communication circuits and switches; a circuit toreceive H.264 streams from high definition video cameras; a circuit tosegment H.264 streams into files and embed retention meta-data in eachheader of a file; a processor core configured to determine that anextrinsic sensor measurement or event is of substantial significance toa surveillance system; and a circuit to set a retention flag in anextrinsic retention meta-data store, wherein said purging modulecomprises: a retention policy store a statutory and judicialrequirements store; a metric threshold store; a camera and locationownership store; an interface to disk file management; and a circuit tostore retention metadata; and a calendar module comprising a calendarprocessor configured to match video file segment timestamps withdate-time of purging directives and when true, trigger purging ofsegments unprotected by retention meta-data.
 5. (canceled)
 6. Theapparatus of claim 4 further comprises: a circuit to receive a purgingdirective; and a circuit to remove pointers to video file segment fileswithout a retention flag set.
 7. An apparatus for high definitionsurveillance image storage optimization comprises: a purging module;communicatively coupled to, a camera network interface to a plurality ofhigh definition video cameras, said purging module, and networkinterface interconnected by communication circuits and switches; whereinsaid purging module comprises: a retention policy store; a statutory andjudicial requirements store; a metric threshold store; a camera andlocation ownership store; an interface to disk file management; acircuit to store retention metadata; and a calendar module comprising acalendar processor configured to match video file segment timestampswith date-time of purging directives and when true, trigger purging ofsegments unprotected by retention meta-data.
 8. (canceled)
 9. Anapparatus for high definition surveillance image storage optimizationcomprises: a camera network interface to a plurality of high definitionvideo cameras; and a video stream transformation module (vtransformationmodule); wherein said transformation module comprises: a first digitallogic circuit to receive H.264 streams transmitted from high definitionvideo cameras coupled to said camera network interface; a firstprocessor core configured by instructions to transform a received H.264stream into at least one video segment; communicatively coupled to, asecond digital logic circuit to transmit a file write operation to mediacontroller to store metrics into file headers encoded into anon-transitory media storage device, wherein said processor cores areone of the group: physical, and virtual; and said circuits, interfaces,modules, cores, and media controller interconnected by communicationcircuits and switches.
 10. The apparatus of claim 9 further comprising:a second processor core configured by instructions to annotate saidvideo segment by insertion of at least one metric of acoustic energytransitions; communicatively coupled to said digital logic circuits andinterfaces.
 11. The apparatus of claim 9 further comprising: a thirdprocessor configured by instructions to annotate video segments byinsertion of metrics of visible and non-visible spectrum energytransitions; communicatively coupled to said digital logic circuits andinterfaces.
 12. The apparatus of claim 9 further comprising: a fourthprocessor core configured by instructions to transform video segments ofthe received H.264 video stream into metrics of movement vectors ofpixel blocks; communicatively coupled to said digital logic circuits andinterfaces.
 13. The apparatus of claim 9 further comprising: a fifthprocessor core configured by instructions to transform video segments ofthe received H.264 video stream into JPEG pixel blocks: communicativelycoupled to said digital logic circuits and interfaces.
 14. The apparatusof claim 9 further comprising: a sixth processor core configured byinstructions to transform frequency coefficients of said JPEG pixelblocks into movement metrics; communicatively coupled to said digitallogic circuits and interfaces.
 15. The apparatus of claim 9 furthercomprising: a circuit to set a retention flag when said metric exceeds athreshold; a circuit to receive a purging directive; and a circuit toremove pointers to video file segment files without a retention flagset.
 16. The apparatus of claim 4 further comprising: an interface to atleast one extrinsic sensor coupled to said extrinsic retention meta-datastore to encode in non-transitory media associated by time with videosegments extrinsic events and sensor measurements of at least one ofportal transit, occupancy of rooms, halls, and location space, andphysical access request and actuation.
 17. The apparatus of claim 16further comprising: sensor measurement of at least one of vibration andmovement from accelerometers, temperature and heat, wind speed andhumidity.
 18. The apparatus of claim 16 further comprising: sensormeasurement of at least one of acoustical sound energy, electromagneticsignals from motion, from smoke, from fire, and from CO detectors. 19.The apparatus of claim 16 further comprising sensor measurement of atleast one of intrusion alarm of equipment and intrusion alarm ofperimeter.